docs: document the opt-in response cache (stage 1/5)#405
Draft
zhongxuanwang-nv wants to merge 2 commits into
Draft
docs: document the opt-in response cache (stage 1/5)#405zhongxuanwang-nv wants to merge 2 commits into
zhongxuanwang-nv wants to merge 2 commits into
Conversation
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
github-actions
Bot
added
size:XXL
License DiffCompared against Lockfile license changesLockfile License ChangesRustAdded
Removed
Updated/Changed
NodeAdded
Removed
Updated/Changed
PythonAdded
Removed
Updated/Changed
Status output |
zhongxuanwang-nv
force-pushed
the
docs/response-cache-stage1
branch
from
be2d512 to
fa77ecb
Compare
An opt-in feature of the adaptive plugin (a response_cache config section, not a new plugin kind): managed LLM calls are keyed by a SHA-256 fingerprint of the normalized request and repeats are served from the store — instant, free, reproducible. Buffered and streaming calls share one keyspace; a streamed miss is teed and stored as its aggregated response, a hit replays provider-native chunks. Keying auto-detects the provider surface from the request shape and trusts the decode only where it is faithful: known-lossy shapes and decodes that fail to round-trip fall back to raw-body fingerprinting, which can only cost a miss. Stateful calls (Responses persistence, conversations, containers) and nondeterministic calls under the safety toggle bypass entirely. Stored answers must be complete: non-null errors, non-final statuses, truncated or lossily-collected streams, and upstream failures are never cached — the CLI gateway relays failed upstream replies to the client verbatim while keeping them invisible to the execution chain. The in-memory store is bounded by an honest resident-size budget with oldest-first eviction; the Redis backend runs under hard deadlines and re-checks entry expiry. Everything fails open: any cache error falls through to a live call. Same config surface in Rust, Python, Node, and Go; doctor reports configuration and backend reachability; hit/miss/ bypass marks carry fingerprints and savings, never bodies. Signed-off-by: Zhongxuan Wang <daniewang@nvidia.com>
Adds a Response Cache page to the adaptive plugin docs: when to use it, a plugins.toml quickstart, per-language plugin configuration, the complete-answers-only storage rules with streaming aggregation and native replay, exact-request key normalization, the response_cache mark and doctor surfaces, a field reference, and the unredacted-at-rest caveat for shared Redis deployments. Links the page from the adaptive About and Configuration pages. Signed-off-by: Zhongxuan Wang <daniewang@nvidia.com>
zhongxuanwang-nv
force-pushed
the
docs/response-cache-stage1
branch
from
fa77ecb to
1e7cafa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PLEASE IGNORE THIS PR FOR NOW
Overview
Documents the opt-in response cache on the documentation site: a new Response Cache page in the Adaptive plugin section, written in plain language with runnable examples — when to use it, a
plugins.tomlquickstart, per-language plugin configuration (Python/Node.js/Rust), what gets cached (complete answers only, streaming aggregation and native replay), how cache keys are built, observability (nemo-relay doctor,response_cachemarks), a full field reference, and the unredacted-at-rest caveat for shared Redis deployments. Docs PR 1/5 of the response-cache series.Details
docs/adaptive-plugin/response-cache.mdx(position 5 in the Adaptive plugin section), following the ACG page skeleton: intro → When To Use It →plugins.tomlexample → Plugin Configuration tabs → Manual API tabs → What Gets Cached → Cache Keys → Observability → Fields → Common Validation Failures.docs/adaptive-plugin/about.mdx(coordination list and Pages list) anddocs/adaptive-plugin/configuration.mdx(Component Shape table and area-pages paragraph).fern checkandfern docs broken-links --strict(the only reported broken links are the pre-existing generated API-reference pages, regenerated byjust docs).Where should the reviewer start?
docs/adaptive-plugin/response-cache.mdx— the page is the whole change (one commit on top of #404).Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)